Techniques for modeling temporal distortions when predicting perceptual video quality

ABSTRACT

In various embodiments, a prediction application computes a quality score for re-constructed visual content that is derived from visual content. The prediction application generates a frame difference matrix based on two frames included in the re-constructed video content. The prediction application then generates a first entropy matrix based on the frame difference matrix and a first scale. Subsequently, the prediction application computes a first value for a first temporal feature based on the first entropy matrix and a second entropy matrix associated with both the visual content and the first scale. The prediction application computes a quality score for the re-constructed video content based on the first value, a second value for a second temporal feature associated with a second scale, and a machine learning model that is trained using subjective quality scores. The quality score indicates a level of visual quality associated with streamed video content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of the co-pending U.S. patentapplication titled, “TECHNIQUES FOR MODELING TEMPORAL DISTORTIONS WHENPREDICTING PERCEPTUAL VIDEO QUALITY,” filed on Feb. 7, 2018 and havingSer. No. 15/890,709. The subject matter of this related application ishereby incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention relate generally to videotechnology and, more specifically, to techniques for modeling temporaldistortions when predicting perceptual video quality.

Description of the Related Art

Efficiently and accurately encoding source video content is critical forreal-time delivery of high-quality video content. Because of variationsin encoded video content quality, it is desirable to implement qualitycontrols to ensure that the visual quality of re-constructed videocontent decoded from the encoded source video content is acceptable.Manually verifying the visual quality of re-constructed video content isprohibitively time consuming. Accordingly, automated video contentquality assessment is often an integral part of the encoding andstreaming infrastructure. For example, automated video qualityassessment can be employed when evaluating encoders and fine-tuningstreaming bitrates to optimize streamed video content.

Some approaches to automatically assessing the quality of re-constructedvideo content involve computing a quality score based on a perceptualquality model that is trained based on human assessments of visualquality. In general, perceptual quality models associate values forfeatures of re-constructed video content with different quality scores.One limitation of predicting the quality of re-constructed video contentvia a perceptual quality model is that perceptual quality modelstypically lack robustness. For example, if none of the featuresassociated with a perceptual quality model reflect certain types ofdistortions, then the perceptual quality model is unable to accuratelypredict human perception of the quality of re-constructed video contentthat include such distortions.

In particular, perceptual quality models are typically unable toreliably predict the perceived visual quality of re-constructed videocontent included in the public-domain LIVE Video Quality Database.Analysis of the deficiencies of perceptual quality models with respectto the LIVE Video Quality Database indicate that none of the featuresassociated with the perceptual quality model accurately model theperceived quality degradation attributable to many types of complextemporal video distortions. Examples of complex temporal videodistortions include, without limitation, motion estimation mismatches,flicker, and ghosting.

As the foregoing illustrates, what is needed in the art are moreeffective techniques for predicting the perceived quality ofre-constructed video content.

SUMMARY OF THE INVENTION

One embodiment of the present invention sets forth acomputer-implemented method for predicting the perceived video qualityof re-constructed video content. The method includes generating a framedifference matrix based on a first frame and a second frame that areincluded in re-constructed video content; computing a first entropymatrix based on the frame difference matrix and a first scale; computinga first value for a first temporal feature based on the first entropymatrix and a second entropy matrix associated with both video contentfrom which the re-constructed video content is derived and the firstscale; and computing a quality score for the re-constructed videocontent based on the first value, a second value for a second temporalfeature associated with a second scale, and a machine learning modeltrained using a plurality of subjective quality scores, wherein thequality score indicates a level of visual quality associated withstreamed video content.

At least one advantage of the disclosed techniques for predictingperceived video quality is that the quality score accurately predictsperceived video quality across a diverse range of re-constructed videocontent. In particular, at least one technological improvement relativeto the prior art is that the disclosed techniques compute, at multiplescales, temporal entropy differences between the video content and theassociated re-constructed video content. Because these type ofcomputations emulate aspects of how the human visual system processesvisual information, the corresponding temporal features accurately modelperceived quality degradation attributable to many types of complextemporal video distortions.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is a conceptual illustration of a system configured to implementone or more aspects of the present invention;

FIG. 2 is a more detailed illustration of the temporal extractor of FIG.1 , according to various embodiments of the present invention;

FIG. 3 is a flow diagram of method steps for computing a quality scorefor re-constructed video content, according to various embodiments ofthe present invention;

FIG. 4 is a conceptual illustration of an ensemble system configured toimplement one or more aspects of the present invention; and

FIG. 5 is a flow diagram of method steps for computing an overallquality score for re-constructed video content based on two differentsets of features, according to various embodiments of the presentinvention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the present invention. However,it will be apparent to one of skilled in the art that the presentinvention may be practiced without one or more of these specificdetails.

The disclosed techniques efficiently and reliably predict quality scoresfor re-constructed video content via a perceptual quality model thatassociates values for features with a quality score. Notably, theperceptual quality model is associated with multiple temporal features,where each temporal feature is associated with a different scale. Inoperation, a temporal extractor computes per-frame values for a specifictemporal feature based on re-constructed video content, the associatedvideo content, and the scale associated with the temporal feature. For agiven frame, the temporal extractor computes a frame difference matrixspecifying differences in luminance between the frame and an adjacentframe. The temporal extractor then down-samples the frame differencematrix based on the scale to generate a scaled frame difference matrix.Subsequently, the temporal extractor performs temporal filteringoperations and local mean-subtraction operations on the scaled framedifference matrix to compute a local mean-subtracted matrix. Thetemporal extractor then performs conditioning operations on the scaledframe difference matrix to generate a first weighted entropy matrix thatis associated with the frame of the re-constructed video content.

In an analogous fashion, the temporal extractor derives a secondweighted entropy matrix based on the video content and the scale. Thetemporal extractor performs subtraction operations between the firstweighted entropy matrix and the second weighted entropy matrix tocompute an entropy difference matrix. Finally, the temporal extractorperforms spatial pooling operations on the entropy difference matrix togenerate a single value for the temporal feature of the frame of there-constructed video content. Because computing entropy differencematrices at multiple scales emulates aspects of how the human visualsystem processes visual information, the perceptual quality modelaccurately models perceived quality degradation attributable to manytypes of complex temporal video distortions.

In some embodiments, an ensemble prediction engine computes a ensemblequality score for re-constructed video content based on two perceptualquality models. Each of the perceptual quality models associates valuesfor a different set of features with a different quality score. Anensemble prediction engine composites quality scores obtained via thetwo perceptual quality models to obtain an ensemble quality score.

Advantageously, distributing features between two prediction qualitymodels decreases the likelihood that increasing the total number offeatures results in inaccuracies attributable to overfitting.

System Overview

FIG. 1 is a conceptual illustration of a system 100 configured toimplement one or more aspects of the present invention. As shown, thesystem 100 includes, without limitation, any number of compute instances110, a training database 120, an objective database 150, a subjectivedatabase 182, and a target feature database 170. For explanatorypurposes, multiple instances of like objects are denoted with referencenumbers identifying the object and parenthetical numbers identifying theinstance where needed. In various embodiments, any number of thecomponents of the system 100 may be distributed across multiplegeographic locations or included in one or more cloud computingenvironments (i.e., encapsulated shared resources, software, data, etc.)in any combination.

As shown, each of the compute instances 110 includes, withoutlimitation, a processor 112 and a memory 116. The processor 112 may beany instruction execution system, apparatus, or device capable ofexecuting instructions. For example, the processor 112 could comprise acentral processing unit (CPU), a graphics processing unit (GPU), acontroller, a microcontroller, a state machine, or any combinationthereof. The memory 116 stores content, such as software applicationsand data, for use by the processor 112 of the compute instance 110.

The memory 116 may be one or more of a readily available memory, such asrandom access memory (RAM), read only memory (ROM), floppy disk, harddisk, or any other form of digital storage, local or remote. In someembodiments, a storage (not shown) may supplement or replace the memory116. The storage may include any number and type of external memoriesthat are accessible to the processor 112. For example, and withoutlimitation, the storage may include a Secure Digital Card, an externalFlash memory, a portable compact disc read-only memory (CD-ROM), anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing.

In general, the compute instances 110 are configured to implement one ormore applications. For explanatory purposes only, each application isdepicted as residing in the memory 116 of a single compute instance 110and executing on a processor 112 of the single compute instance 110.However, as persons skilled in the art will recognize, the functionalityof each application may be distributed across any number of otherapplications that reside in the memories 116 of any number of computeinstances 110 and execute on the processors 112 of any number of computeinstances 110 in any combination. Further, the functionality of anynumber of applications may be consolidated into a single application orsubsystem.

In particular, the compute instances 110 are configured to provideautomated video content quality assessment for re-constructed videocontent. The video content quality assessment may be used to implementquality controls associated with an encoding and streaminginfrastructure. For example, to ensure an acceptable viewing experience,the quality of re-constructed video content and, indirectly, the caliberof the elements included in the encoding and streaming infrastructureare typically evaluated at various points in the design and deliveryprocess based on visual quality scores. The visual quality scores arethen used to guide the development of applications (e.g., encoders) andthe real-time optimization of content delivery, such as stream-switchingalgorithms that are quality-aware.

As shown, a training feature engine 130, a training engine 180, aprediction engine 192, an instance of a feature extractor 140, and atemporal pooling engine 194 reside in the memory 116 and execute on theprocessor 112 of different compute instances 110. In alternateembodiments, any number of instances of the training feature engine 130,the training engine 180, the feature extractor 140, the predictionengine 192, the temporal pooling engine 194, the training feature engine130, and the training engine 180 may reside in any number and types ofmemory and execute on any number and types of processing devices in anycombination.

During a learning phase, depicted with dotted lines, the trainingfeature engine 130 and the training engine 180 collaborate to generate aperceptual quality model 190. For re-constructed video content, theperceptual quality model 190 associates a set of values for a set offeatures with a quality score 184 that correlates to subjective humanvisual experience when viewing the re-constructed video content.

In operation, the training feature engine 130 acquires a trainingdatabase 120 and generates the objective database 150. The trainingdatabase 120 includes, without limitation, any number of sources 122,and any number of re-constructed sources 124. Each of the sources 122includes, without limitation, any amount and type of video content.Examples of video content include, without limitation, any portion(including all) of feature length films, episodes of televisionprograms, and music videos, to name a few. Each of the re-constructedsources 124 include, without limitation, re-constructed video contentthat is derived from the video content included in one of the sources122.

As a general matter, each of the re-constructed sources 124 replicatesthe video content included in the associated source 122 as conveyed to aviewer via a streaming infrastructure and a display device. There-constructed sources 124 may be derived from the sources 122 based onany number of encoding parameters and in any technically feasiblefashion. For instance, in some embodiments, three differentre-constructed sources 124 are derived from each of the sources 122. Foreach of three different sets of encoding parameters, an encoding engine(not shown in FIG. 1 ) independently encodes the source 122(x). For eachof the three resulting encoded sources, a decoding engine (not shown inFIG. 1 ) independently decodes the encoded source to generate one of there-constructed sources 124 derived from the sources 122(x).

As shown, the training feature engine 130 includes, without limitation,the feature extractor 140. The feature extractor 140 computes values forfeatures. Each of the features is either a “spatial feature” thatmeasures spatial video distortions or a “temporal feature” that measurestemporal visual distortions. Examples of spatial video distortionsinclude, without limitation, compression artifacts, scaling artifacts,and film grain noise. Examples of temporal video distortions include,without limitation, motion estimation mismatches, flicker, and ghosting.

The feature extractor 140 includes, without limitation, a spatialextractor 148 and a temporal extractor 146. For each re-constructedsource 124(x) and each spatial feature, the spatial extractor 148computes a spatial array 172(x) based on the re-constructed source124(x) and, optionally, the source 122 associated with there-constructed source 124(x). The spatial array 172(x) includes, withoutlimitation, a different frame value for each frame included in there-constructed source 124(x).

Similarly, for each re-constructed source 124(x) and each temporalfeature, the temporal extractor 146 computes a temporal array 174(x)based on the re-constructed source 124(x) and the source 122 associatedwith the re-constructed source 124(x). The temporal array 174(x)includes, without limitation, a different frame pair value for eachframe included in the re-constructed source 124(x). As referred toherein, a “frame pair” comprises two sequential frames, and the framepair value is associated with the first of the two sequential frames. Inalternate embodiments, the feature extractor 140 may compute any numberof values for a particular feature of a particular re-constructed source124 in any technically feasible fashion.

For each spatial feature and each re-constructed source 124(x), thetraining feature engine 130 performs temporal pooling operations on thespatial array 172(x) associated with the spatial feature to determine aspatial value 162 for the feature of the re-constructed source 124(x).The training feature engine 130 may perform any amount and type oftemporal pooling operations in any technically feasible fashion. Forinstance, in some embodiments, the training feature engine 130 could setthe spatial value 162 for a given spatial feature of the re-constructedsource 124(x) equal to the average of the frame values included in thespatial array 172(x) associated with the feature. Similarly, for eachtemporal feature and each re-constructed source 124(x), the trainingfeature engine 130 could perform set the temporal value 164 for a giventemporal feature of the re-constructed source 124(x) equal to theaverage of the frame pair values included in the temporal array 174(x)associated with the feature.

For each of the re-constructed sources 124, the training feature engine130 generates a training feature database 152 that includes, withoutlimitation, the spatial values 162 and the temporal values 164 for thefeatures of the re-constructed source 124. The training feature engine130 then generates the objective database 150 that includes, withoutlimitation, the training feature databases 152. Consequently, if thetraining database 120 includes N re-constructed sources 124, then theobjective database 150 includes N training feature databases 152.Further, if the feature extractor 140 computes S spatial arrays 172 forS spatial features and T temporal arrays 174 for T temporal features,then each of the training feature databases 152 includes S spatialvalues 162 and T temporal values 164.

The training engine 180 generates the perceptual quality model 190 basedon the objective database 150 and the subjective database 182. Thesubjective database 182 includes, without limitation, a quality score184 of each of the re-constructed sources 142. The quality score 184(x)for the re-constructed source 142(x) is generated based on humanviewings of the re-constructed source 142(x). More specifically, asviewers watch the re-constructed source 142(x) on display devices, theviewers personally rate the visual quality—assigning personalizedscores. The personalized scores are then aggregated to generate thequality score 184(x).

The training engine 180 implements any number and type of machinelearning algorithms in any technically feasible fashion to generate theperceptual quality model 190 based on the objective database 150 and thesubjective database 182. Examples of machine learning algorithmsinclude, without limitation, support vector machine algorithms,artificial neural network algorithms, and random forest algorithms, toname a few. In general, for each of the re-constructed sources 124(x),the perceptual quality model 190 is trained based on the associationbetween the spatial values 162 and the temporal values 164 included inthe objective database 150(x) and the quality score 184(x) included inthe subjective database 182. In alternate embodiments, the trainingengine 180 may train the perceptual quality model 190 at any technicallyfeasible level of granularity (e.g., based on associating frame valuesfor the features with frame quality scores).

Notably, irrespective of the granularity at which the training engine180 trains the perceptual quality model 190, the perceptual qualitymodel 190 associates a set of values for the set of features at anygiven granularity to a score at the same granularity. More precisely,the perceptual quality model 190 associates a set of S values for theset of S spatial features and a set of T values for the set of Ttemporal features with a score that predicts an associated perceivedvisual quality. For example, the perceptual quality model 190 associatesa set of S frame values for the spatial features of the first frame anda set of T frame pair values for the temporal features of the firstframe to a frame score that represents the perceived visual quality ofthe first frame.

During a prediction phase, depicted with solid lines, the featureextractor 140, the prediction engine 192, and the temporal poolingengine 194 collaborate to predict the quality score 184 of are-constructed target 144 based on the perceptual quality model 190, there-constructed target 144, and a target 142. The feature extractor 140,the prediction engine 192, and the temporal pooling engine 194 arecollectively referred to herein as a “prediction application.” There-constructed target 144 includes, without limitation, re-constructedvideo content that is derived from the video content included in thetarget 142. The target 142 includes, without limitation, any amount andtype of video content.

The re-constructed target 144 replicates the video content included inthe associated source 122 as conveyed to a viewer via a streaminginfrastructure and a display device. The re-constructed target 144 maybe generated in any technically feasible fashion. For instance, in someembodiments, an encoder encodes the target 142 based on a constant rateencoding strategy to generate an encoded target. Subsequently, a decoderdecodes the encoded target to generate the re-constructed target 144.

For each spatial feature associated with the perceptual quality model190, the spatial extractor 148 computes the spatial array 172 based onthe re-constructed target 144 and, optionally, the target 142. For eachtemporal feature associated with the perceptual quality model 190, thetemporal extractor 146 computes the temporal array 174 based on there-constructed target 144 and the target 142. In this fashion, for eachframe included in the re-constructed target 144, the feature extractor140 computes a different frame value for each spatial feature and adifferent frame pair value for each temporal feature.

For each frame included in the re-constructed target 144, the predictionengine 192 generates a frame score based the frame values associatedwith the frame, the frame pair values associated with the frame, and theperceptual quality model 190. For example, for the first frame includedin the re-constructed target 144, the prediction engine 192 determines afirst set of values for the features based on the first entries of thespatial arrays 172 and the first entries of the temporal arrays 174. Theprediction engine 192 then determines the frame score for the firstframe based on the first set of values for the features and theperceptual quality model 190. In alternate embodiments, the predictionengine 192 may compute scores for perceived visual quality at any levelof granularity based on values for the features at the same level ofgranularity and the perceptual quality model 190.

Subsequently, the temporal pooling engine 194 determines the qualityscore 184 of the re-constructed target 144 based on the arithmetic meanof the frame scores. More specifically, the temporal pooling engine 184sums all of the frame scores and then divides the summation by the totalnumber of frames to compute the quality score 184. Accordingly, thequality score 184 represents an average visual quality across the framesincluded in the re-constructed target 144. Finally, the temporal poolingengine 194 transmits or displays the quality score 184 of there-constructed target 144 for evaluation of streamed video content.

In alternate embodiments, the temporal pooling engine 194 may determinethe quality score 184 of the re-constructed target 144 based on theframe scores in any technically feasible fashion. For instance, in someembodiments, the temporal pooling engine 194 may perform hysteresispooling operations that mimic the relatively smooth variance of humanopinion scores in response to changes in video quality. More precisely,the temporal pooling engine 194 could perform both linear low passoperations and non linear (rank-order) weighting operations on the framescores to generate the quality score 184 of the re-constructed target144.

In some alternate embodiments, the temporal pooling engine 194determines a different quality score 184 of each chunk of there-constructed target 144 instead of a single quality score 184 for there-constructed target 144. As referred to here, a “chunk” typicallycorresponds to a scene, and a scene is a sequence of frames were thespatiotemporal characteristics are approximately constant. To computethe quality score 184 of a particular chunk of the re-constructed target144, the temporal pooling engine 194 computes the arithmetic mean of theframe scores for the subset of the frames that are included in thechunk.

One limitation of predicting the quality of re-constructed video contentvia a conventional perceptual quality model is that conventionalperceptual quality models typically lack robustness. For example, ifnone of the features associated with a conventional perceptual qualitymodel reflect certain types of distortions, then the conventionalperceptual quality model is unable to accurately predict humanperception of the quality of re-constructed video content that includesuch distortions.

A conventional perceptual quality model is typically associated withmultiple spatial features and a single temporal feature. The spatialfeatures may include without limitation, any number of additiveimpairment measures (ADM) features, blind or referenceless image spatialquality evaluator features, and visual information fidelity (VIF)features, to name a few. The temporal feature is a frame differencefeature. For a given frame pair, a conventional temporal extractorcomputes per-pixel differences in luminance between the two frames. Thetemporal extractor then sets the frame pair value for the framedifference feature equal to the average of the per-pixel differences inluminance.

However, as demonstrated by benchmarking conducted using thepublic-domain LIVE Video Quality Database, the frame difference featuredoes not accurately measure the perceived quality degradationattributable to many types of complex temporal video distortions.Examples of complex temporal video distortions include, withoutlimitation, motion estimation mismatches, flicker, and ghosting. Becausethe frame difference feature is an unreliable measure of complextemporal video distortions, conventional perceptual quality models areunable to robustly predict the perceived visual quality ofre-constructed video content.

Implementing Accurate Temporal Features

To address the above problems, the system 100 establishes multipletemporal features that emulate aspects of how the human visual systemprocesses visual information. For instance, as described in conjunctionwith FIG. 2 , because the human visual system processes visualinformation at multiple scales, the temporal extractor 146 computesframe pair values for multiple temporal features, where each temporalfeature is associated with a different scale. In alternate embodiments,the temporal extractor 146 may compute frame pair values for any numberof temporal features in any technically feasible fashion that ensuresthat multiple scales are represented. In general, the temporal extractor146 computes frame pair values for temporal features that areperceptually relevant and, together, accurately measure perceivedquality degradation attributable to complex temporal video distortions.

It will be appreciated that the system 100 shown herein is illustrativeand that variations and modifications are possible. For example thefunctionality provided by the training feature engine 130, the trainingengine 180, the feature extractor 140, the spatial extractor 148, thetemporal extractor 146, the prediction engine 192, and the temporalpooling engine 194 as described herein may be integrated into ordistributed across any number of software applications (including one)and any number of components of the system 100. Further, the connectiontopology between the various units in FIG. 1 may be modified as desired.

FIG. 2 is a more detailed illustration of the temporal extractor 146 ofFIG. 1 , according to various embodiments of the present invention. Asshown, the temporal extractor 146 includes, without limitation, a framecomparison engine 220, a down-sampling engine 240, a spatial filteringengine 250, an entropy engine 270, an entropy comparison engine 280, anda spatial pooling engine 290.

In general, the temporal extractor 146 generates values for the temporalfeatures of re-constructed video content based on the re-constructedvideo content and the video content from which the re-constructed videocontent is derived. Notably, each of the temporal features is associatedwith a different scale 250. The re-constructed video content may be anyof the re-constructed sources 124 or the re-constructed target 144. Forexplanatory purposes only, the temporal extractor 146 is described inFIG. 2 in the context of generating the temporal array 174(1) for afirst temporal feature of the re-constructed target 144.

As shown, the re-constructed target 144 includes, without limitation,frames 210(1)-210(F), where F is any integer greater than 1. In acomplementary fashion, the target 142 includes, without limitation, theframes 210′(1)-210′(F). For explanatory purposes only, an object that isassociated with the target 142 is distinguished from a comparable objectthat is associated the re-constructed target 144 with a prime symbol(i.e., ′) that decorates the reference number. As described inconjunction with FIG. 1 , each of the re-constructed target 144 and thetarget 142 are associated with F frames 210 and F frame pairs.

In operation, the frame comparison engine 220(1) computes framedifference matrices 230(1)-230(F) based on the re-constructed target144. Each frame difference matrix 230 is associated with a differentframe pair. In general, to compute the frame difference matrix 230(x),the frame comparison engine 220(1) performs per-pixel comparisonoperations between the frames 210(x) and 210(x+1) to determine per-pixelluminance differences. Accordingly, each pixel location included in theframe difference map 230(x) specifies a different per-pixel luminancedifference. In an analogous manner, the frame comparison engine 220(2)independently computes the frame difference matrices 230′(1)-230′(F)based on the target 142.

The down-sampling engine 240(1) independently performs down-samplingoperations on each of the frame difference matrices 230(1)-230(F) basedon the scale 250(1) associated with the first temporal feature togenerate, respectively, scaled frame difference (FD) matrices242(1)-242(F). As shown, the scale 250(1) is equal to 2. In alternateembodiments, the first temporal feature may be associated with any scale250 in any technically feasibly fashion. The down-sampling engine 240(1)may perform any number and type of down-sampling operations based on thescale 250(1). In general, if each of the frame difference matrices 230includes N by M pixels and the scale 240(1) is Q, then each of thescaled frame difference matrices 242 includes N/Q by M/Q pixels. In ananalogous manner, the down-sampling engine 240(2) independently performsdown-sampling operations on each of the frame difference matrices230′(1)-230′(F) based on the scale 240(1) to generate, respectively, thescaled frame difference matrices 242′(1)-242′(F).

The spatial filtering engine 250(1) independently processes each of thescaled frame difference matrices 242(1)-242(F) to generate,respectively, local mean-subtracted matrices 262(1)-262(F). To processthe scaled frame difference matrix 242(x), the spatial filtering engine250(1) performs a two-step localized spatial filtering process thatapproximates bandpass filtering of the scaled frame difference matrix242(x). First, the spatial filtering engine 250(1) filters the scaledframe difference matrix 242(x) with a spatial isotropic Gaussian filterto generate a locally averaged scaled frame difference matrix (notshown). The spatial filtering engine 250(1) then subtracts the locallyaveraged scaled frame difference matrix from the scaled frame differencematrix 242(x) to generate the local mean-subtracted matrix 262(x). In ananalogous fashion, the spatial filtering engine 250(2) independentlyprocesses each of the scaled frame difference matrices 242′(1)-242′(F)to generate, respectively, the local mean-subtracted matrices262′(1)-262′(F).

The entropy engine 270(1) independently performs conditioning operationson each of the local mean-subtracted matrices 262(1)-262(F) to generate,respectively, weighted entropy matrices 272(1)-272(F). Notably, theconditioning operations are related to divisive normalization processthat is known to occur in the primary visual cortex (area V1). Toprocess the local mean-subtracted matrix 262(x), the spatial filteringengine 250(1) assigns the samples included in the local mean-subtractedmatrix 262(x) to overlapping blocks, where each block includes a b×b setof neighboring samples. The spatial filtering engine 250(1) thenestimates a b×b covariance matrix based on the overlapping blocks. Thespatial filtering engine 250(1) also partitions the samples included inthe local mean-subtracted matric 282(x) into non-overlapping b×b blocks.For each non-overlapping block, the spatial filtering engine 250(1)computes the variance of the non-overlapping block.

Subsequently, for each non-overlapping block, the spatial filteringengine 250(1) computes an entropy value for the non-overlapping blockbased on the covariance matrix. Finally, for each non-overlapping block,the spatial filtering engine 250(1) weighs the entropy value of thenon-overlapping block by the variance of the non-overlapping block via alogarithmic function to compute a weighted entropy value. Accordingly,the weighted entropy matrix 272(x) includes, without limitation, aweighted entropy value for each non-overlapping block. In an analogousfashion, the spatial filtering engine 250(2) independently processeseach of the local mean-subtracted matrices 262′(1)-262′(F) to generate,respectively, the weighted entropy matrices 272′(1)-272′(F).

The entropy difference engine 280 computes entropy difference matrices282(1)-282(F) based on the weighted entropy matrices 272(1)-272(F)associated with the re-constructed target 144 and the weighted entropymatrices 272′(1)-272′(F) associated with the target 142. Morespecifically, to compute the entropy difference matrix 282(x), theentropy difference engine 280 computes the difference between each ofthe weighted entropy values included in the weighted entropy matrix272(x) and the corresponding weighted entropy value included in theweighted entropy matrix 272′(x). Accordingly, the entropy differencematrix 282(x) includes, without limitation, an “entropy differencevalue” for each non-overlapping block.

The spatial pooling engine 290 performs spatial pooling operations togenerate the temporal array 174(1) based on the entropy differencematrices 282. As shown, the temporal array 174(1) includes, withoutlimitation, frame pair values 264(1)-264(x). Each of the frame pairvalues 264 included in the temporal array 174(1) specifies a value forthe first temporal feature of a different one of the frame pairsincluded in the re-constructed target 144. To compute the frame pairvalue 264(x), the spatial pooling engine 290 averages the entropydifference values over all the non-overlapping blocks included in theentropy difference matrix 282(x).

In alternate embodiments, the spatial pooling engine 290 may aggregatethe entropy difference values included in the entropy difference matrix282(x) in any technically feasible fashion to generate the frame pairvalue 264(x). Advantageously, entropy differences between re-constructedvideo content and the video content from which the re-constructed videocontent is derived correlate with subjective human assessments of thequality of the re-constructed video content.

Notably, a second temporal feature is associated with the scale 250(2)of 3 and a third temporal feature is associated with the scale 240(3) of4. When computing the temporal array 174(2), the down-sampling engines240(1) and 240(2) perform down-sampling operations based on the scale240(2) of 3. When computing the temporal array 174(3), the down-samplingengines 240(1) and 240(2) perform down-sampling operations based on thescale 240(3) of 4. The operations of the frame comparison engine 220,the spatial filtering engine 250, the entropy engine 270, the entropycomparison engine 280, and the spatial pooling engine 290 are notaltered based on the scale 250. Advantageously, computing temporalfeatures at multiple scales reflects multiscale processing inherent inthe human vision system.

Note that the techniques described herein are illustrative rather thanrestrictive, and may be altered without departing from the broaderspirit and scope of the invention. Many modifications and variations onthe functionality provided by the temporal extractor 146, the framecomparison engine 220, the down-sampling engine 240, the spatialfiltering engine 250, the entropy engine 270, the entropy comparisonengine 280, and the spatial pooling engine 290 will be apparent to thoseof ordinary skill in the art without departing from the scope and spiritof the described embodiments.

For instance, in various embodiments, the temporal extractor 146 may beconfigured to compute values for any number of temporal feature based onany of the matrices at any number of scales 250. For example, thetemporal extractor 146 could compute frame pair values for a first framedifference feature based on the frame difference matrices 230 and 230′.The temporal extractor 146 could compute frame pair values for a secondframe difference feature based on the frame difference matrices 242 and242′ that are associated with the scale 250 of 2. The temporal extractorcould compute frame pair values for a first entropy feature based on theentropy difference matrices 282 that are associated with the scale 250of 2. Finally, the temporal extractor could compute frame pair valuesfor a second entropy feature based on the entropy difference matrices282 that are associated with the scale 250 of 3.

FIG. 3 is a flow diagram of method steps for computing a quality scorefor re-constructed video content, according to various embodiments ofthe present invention. Although the method steps are described withreference to the systems of FIGS. 1-2 , persons skilled in the art willunderstand that any system configured to implement the method steps, inany order, falls within the scope of the present invention.

As shown, a method 300 begins at step 302, where the frame comparisonengine 220 generates the frame difference matrices 230 based on there-constructed target 144 and the frame difference matrices 230′ basedon the target 142. At step 304, the temporal extractor 146 selects thefirst temporal feature and the scale 250(1) associated with the firsttemporal feature. At step 306, the down-sampling engine 240 down-samplesthe frame difference matrices 230 and the frame difference matrices 230′based on the selected scale 250 to generate, respectively, the scaledframe difference matrices 242 and the scaled frame difference matrices242′.

At step 308, for each of the scaled frame difference matrices 242, thespatial filtering engine 250 independently performs localized spatialfiltering operations on the frame difference matrix 242 to generate thecorresponding local mean-subtracted matrix 262. In an analogous fashion,for each of the scaled frame difference matrices 242′, the spatialfiltering engine 250 independently performs localized spatial filteringoperations on the scaled frame difference matrix 242′ to generate thecorresponding local mean-subtracted matrix 262′.

At step 310, for each of the local mean-subtracted matrices 262, theentropy engine 270 performs conditioning operation on the localmean-subtracted matrix 262 to compute the weighted entropy matrix 272.In an analogous fashion, for each of the local mean-subtracted matrices262′, the entropy engine 270 performs conditioning operation on thelocal mean-subtracted matrix 262′ to compute the weighted entropy matrix272′.

At step 312, the entropy difference engine 280 compares the weightedentropy matrices 272 associated with the re-constructed target 144 tothe weighted entropy matrices 272′ associated with the target 142 tocompute the entropy difference matrices 282. More specifically, tocompute the entropy difference matrix 282(x), the entropy differenceengine 280 computes the difference between each of the weighted entropyvalues included in the weighted entropy matric 272(x) and thecorresponding weighted entropy value included in the weighted entropymatrix 272′(x). At step 314, for each of entropy difference matrices282, the spatial pooling engine 290 performs spatial pooling operationsto compute the frame pair value 264(x) included in the temporal array174 associated with the selected temporal feature.

At step 316, the temporal extractor 146 determines whether the selectedtemporal feature is the last temporal feature. If, at step 316, thetemporal extractor 146 determines that the selected temporal feature isnot the last temporal feature, then the method 300 proceeds to step 318.At step 318, the temporal extractor 146 selects the next feature and theassociated scale 250, and the method 300 returns to step 306, where thedown-sampling engine 240 down-samples the frame difference matrices 230and the frame difference matrices 230′ based on the selected scale 250.

If, however, at step 316, the temporal extractor 146 determines that theselected temporal feature is the last temporal feature, then the method300 proceeds directly to step 320. At step 320, for each spatialfeature, the spatial extractor 146 computes frame values based on there-constructed target 144 and, optionally, the target 142. At step 322,for each of the frames 210, the prediction engine 192 generate a framescore based on the frame values associated with the frame, the framepair values associated with the frame, and the perceptual quality model190.

At step 324, the temporal pooling engine 194 performs temporal poolingoperations on the frame scores to determine the quality score 184 of there-constructed target 144. The temporal pooling engine 194 may performany number and type of temporal pooling operations to aggregate theframe scores. For instance, in some embodiments, the temporal poolingengine 194 determines the quality score 184 of the re-constructed target144 based on the arithmetic mean of the frame scores. The temporalpooling engine 194 then transmits or displays the quality score 184 ofthe re-constructed target 144 for evaluation of streamed video content.The method 300 then terminates.

Advantageously, any number of the quality scores 184 may be aggregatedto generate an aggregate quality score as part of evaluating the visualquality of streamed media content. For example, a dashboard applicationthat monitors a streaming service on a daily basis can indicate thehealth of the streaming service based on the aggregated quality score.In general, unless a change occurs to the streaming service, theaggregated quality score remains stable. Consequently, a deviation inthe aggregated quality score between a particular day and theimmediately preceding day indicates a change to the streaming service(e.g., an issue or an improvement) that impacts the visual quality ofthe streamed media content. By contrast, if a change to the streamingservice is not intended to affect the visual quality of the streamedmedia content, then the aggregated quality score can provideconfirmation that the visual quality of the streamed media content isunaffected. For instance, if a new encode for a smart phone savesbandwidth while preserving visual quality, then the aggregated qualityscore remains stable while a time-weighted bitrate decreases.

In another example, an A/B test application compares a control group ofstreaming sessions with one or more treatment groups of streamingsessions. A treatment group differs from the control group in exactlyone key parameter. The aggregated quality score indicates whether thevisual quality of the streamed media content is impacted by thedifference in the key parameter. For instance, if the treatment groupuses a new encode and the control group uses a current encode, then theA/B test application could compare the aggregated quality score for thecontrol group and the aggregated quality score for the treatment group.The results of the comparison indicate whether the new encode impactsthe visual quality of streamed media content for the treatment groups.

Predicting Visual Quality Based on Complementary Perceptual QualityModels

As persons skilled in the art will recognize, as the number of featuresassociated with the perceptual quality model 190 increases, thelikelihood of overfitting the perceptual quality model 190 alsoincreases. When the perceptual quality model 190 is overfit, theperceptual quality model 190 models idiosyncrasies in the training data(e.g., the objective database 150 and the subjective database 184).Consequently, an overfit perceptual quality model 190 is often unable toaccurately predict the quality score 184 of re-constructed video contentthat is not included in the training database 120. To reduce thelikelihood of overfitting a single perceptual quality model 190, someembodiments of the present invention implement multiple, complementaryperceptual quality models 190.

FIG. 4 is a conceptual illustration of an ensemble system 400 configuredto implement one or more aspects of the present invention. As shown, theensemble system 400 includes, without limitation, any number of thecompute instances 110. As shown, an ensemble training engine 410 residesin the memory 112 and executes on the processor of one of the computerinstances 110. An ensemble prediction engine 490 resides in the memory112 and execute on the processor 116 of another computer instance 110.In alternate embodiments, the ensemble training engine 410 and theensemble prediction engine 490 may reside in any number and type ofmemories and execute on any number and type of processing devices in anycombination. The ensemble prediction engine 490 is also referred toherein as an “ensemble prediction application.”

The ensemble training engine 410 includes without limitation, a featureassigner 420 and two instances of the training engine 180. The featureassigner 420 receives an overall feature set 412 that includes anynumber of features and generates two features sets 422(1) and 422(2).More precisely, the feature assigner 420 assigns each of the featuresincluded in the overall feature set 412 to at least one of the featuressets 422.

The feature assigner 420 may be configured to assign each of thefeatures included in the overall feature set 412 to at least one of thefeatures sets 422 in any technically feasible fashion. For example, insome embodiments, the feature assigner 420 generates the features sets422 based on user input received via a graphical user interface (notshown). In other embodiments, the feature assigner 420 generates thefeature sets 422 based on any number of heuristics that optimize anynumber and type of criteria.

For explanatory purposes only, the overall feature set 412 depicted inFIG. 4 includes five spatial features and four temporal features. Thefive spatial features include one VIF feature and four ADM features(associated with four different scales 250). The four temporal featuresinclude one frame difference feature and three entropy features(associated with three different scales 250). The feature assigner 420assigns the five spatial features to both the feature set 422(1) and thefeature set 422(2), the frame difference feature to the feature set422(1), and the three entropy features to the feature set 422(2).

Subsequently, the ensemble training engine 410 configures the trainingengine 180(1) to generate the perceptual quality model 190(1) based onthe subjective database 182 and the portion of the objective database150 that specifies values for the features included in the feature set422(1). By contrast, the ensemble training engine 410 configures thetraining engine 180(2) to generate the perceptual quality model 190(2)based on the subjective database 182 and the portion of the objectivedatabase 150 that specifies values for the features included in thefeature set 422(2).

In alternate embodiments, the ensemble training engine 410 includes twodifferent training engines that implement different machine learningalgorithms. The ensemble training engine 410 configures the firsttraining engine to generate the perceptual quality model 190(1) based onthe subjective database 182 and the portion of the objective database150 that specifies values for the features included in the feature set422(1). The ensemble training engine 410 configures the second trainingengine to generate the perceptual quality model 190(2) based on thesubjective database 182 and the portion of the objective database 150that specifies values for the features included in the feature set422(2).

In general, the perceptual quality model 190(1) associates a set ofvalues for the feature set 422(1) with a first quality score, and theperceptual quality model 190(2) associates a set of values for thefeature set 422(2) with a second quality score. Advantageously, becauseneither of the perceptual quality models 190 is associated with all ofthe features included in the overall feature set 422, the likelihoodthat the ensemble training engine 410 generates overfit perceptualquality models 190 is mitigated.

The ensemble prediction engine 490 includes, without limitation, twoinstances of the prediction engine 192(1) and 192(2), an aggregationengine 440, and the temporal pooling engine 194. The ensemble predictionengine 490 receives the perceptual quality model 190(1), the perceptualquality model 190(2), and the target feature database 170 associatedwith the re-constructed target 144. In alternate embodiments, theensemble prediction engine 490 configures the feature extractor 140 tocompute the target feature database 170.

The ensemble prediction engine 490 configures the prediction engine192(1) to compute a frame prediction database 430(1) based on theperceptual quality model 190(1) and the portion of the target featuredatabase 170 associated with the features included in the feature set422(1). For each frame 210(x) included in the re-constructed target 144,the frame prediction database 430(1) includes, without limitation, aframe score 434(x) that represents a predicted perceived visual qualityof the frame 210. In operation, for the frame 210(x), the ensembleprediction engine 490 determines a set of values for the feature set422(1) based on the target feature database 170. The ensemble predictionengine 490 then configures the prediction engine 192(1) to compute theframe score 434(x) included in the frame prediction database 430(1)based on the set of values for the feature set 422(1) and the perceptualquality model 190(1).

In a complementary fashion, the ensemble prediction engine 490configures the prediction engine 192(2) to compute the frame predictiondatabase 430(2) based on the perceptual quality model 190(2) and theportion of the target feature database 170 associated with the featuresincluded in the feature set 422(2). In operation, for the frame 210(x),the ensemble prediction engine 490 determines a set of values for thefeature set 422(2) based on the target feature database 170. Theensemble prediction engine 490 then configures the prediction engine192(2) to compute the frame score 434(x) included in the frameprediction database 430(2) based on the set of values for the featureset 422(2) and the perceptual quality model 190(2).

Upon receiving the frame prediction database 430(1) and the frameprediction database 430(2), the aggregation engine 440 generates anaggregated prediction database 450 that includes a different aggregatedframe score 454 for each of the frames 210. More precisely, theaggregation engine 440 performs any number of aggregation operations onthe frame score 434(x) included in the frame prediction database 430(1)and the frame score 434(x) included in the frame prediction database430(2) to generate the aggregated frame score 454(x) for the frame210(x).

The aggregation engine 440 may compute the aggregated frame scores 454in any technically feasible fashion. For instance, in some embodiments,the aggregation engine 440 computes the average of the frame score434(x) included in the frame prediction database 430(1) and the framescore 434(x) included in the frame prediction database 430(2) togenerate the aggregated frame score 454(x) for the frame 210(x). Inother embodiments, the aggregation engine 440 computes a weightedaverage of the frame score 434(x) included in the frame predictiondatabase 430(1) and the frame score 434(x) included in the frameprediction database 430(2) to generate the aggregated frame score 454(x)for the frame 210(x).

In yet other embodiments, the aggregation engine 440 may perform anynumber of machine learning algorithms to generate an aggregation modelthat associates a frame score 434 associated with the feature set 422(1)and a corresponding frame score 434 associated with the feature set422(2) with the aggregated frame quality score 454. Examples of machinelearning algorithms that the aggregation engine 440 may perform include,without limitation, bayes-based methods, bagging, boosting, bayesianparameter averaging, bayesian model combination, bucket of models,stacking, and the like. The aggregation model may or may not implementthe same machine learning algorithms that the training engine 180implements to generate the perceptual quality models 190. Subsequently,the aggregation engine 440 may compute the aggregated predictiondatabase 450 that includes the aggregated frame scores 454 based on theaggregation model, the frame prediction database 430(1), and the frameprediction database 430(2).

After the aggregation engine 440 generates the aggregated predictiondatabase 450, the temporal pooling engine 194 performs any number oftemporal pooling operations on the aggregated prediction database 450 tocompute the quality score 184 of the re-constructed target 144. Asdescribed in detail in conjunction with FIG. 1 , some examples oftemporal pooling operations include arithmetic mean operations andhysteresis pooling operations. Further, as also described in detail inconjunction with FIG. 1 , in alternate embodiments, the temporal poolingengine 194 may generate different quality scores 184 for each chunk ofthe re-constructed target 144.

Notably, the reliability of the quality score 184 correlates to thereliability of the perceptual quality model 190(1), the reliability ofthe perceptual quality model 190(2), and the degree to which theperceptual quality models 190 complement each other. Accordingly, thefeature assigner 420 may be configured to generate the features sets 422based on any number of heuristics that optimize the accuracy and/orrobustness of the quality score 184.

For instance, in some embodiments, the feature assigner 420 implementsheuristics that optimize the features included in each of the featuresets 422 to capture spatiotemporal aspects of perceptual visual quality,result in an accurate and robust perceptual quality model 190, and arediverse within the feature set 422 as well as across the feature sets422.

Note that the techniques described herein are illustrative rather thanrestrictive, and may be altered without departing from the broaderspirit and scope of the invention. Many modifications and variations onthe functionality provided by the ensemble training engine 410, thefeature assigner 420, the training engine 180, the ensemble predictionengine 490, the prediction engine 192, the aggregation engine 440, andthe temporal pooling engine 194 will be apparent to those of ordinaryskill in the art without departing from the scope and spirit of thedescribed embodiments.

FIG. 5 is a flow diagram of method steps for computing an overallquality score for re-constructed video content based on two differentsets of features, according to various embodiments of the presentinvention. Although the method steps are described with reference to thesystems of FIGS. 1, 2, and 4 , persons skilled in the art willunderstand that any system configured to implement the method steps, inany order, falls within the scope of the present invention.

As shown, a method 500 begins at step 502, where for each featureincluded in the overall feature set 412, the feature assigner 420assigns the feature to at least one of the feature set 422(1) and thefeature set 422(2). Notably, at least one feature included in each ofthe feature sets 422 is not included in the other feature set 422. Atstep 504, the ensemble training engine 410 configures the trainingengine 180(1) to generate the perceptual quality model 190(1) based onthe subjective database 182 and the portion of the objective database150 associated with the feature set 422(1). At step 506, the ensembletraining engine 410 configures the training engine 180(2) to generatethe perceptual quality model 190(2) based on the subjective database 182and the portion of the objective database 150 associated with thefeature set 422(2).

At step 508, the ensemble prediction engine 490 receives the target 142and the re-constructed target 144. At step 510, for each frame 210included in the re-constructed target 144, the ensemble predictionengine 490 computes a first set of values for the feature set 422(1) anda second set of values for the feature set 422(2). More precisely, theensemble prediction engine 490 configures the feature extractor 140 tocompute the target feature database 170 based on the re-constructedtarget 144 and the target 142. For each of the frames 210, the ensembleprediction engine 490 then identifies the first set of values for thefeature set 422(1) and the second set of values for the feature set422(2) based on the target feature database 170. In alternateembodiments, for each of the frames 210, the ensemble prediction engine490 may compute the first set of values for the feature set 422(1) andthe second set of values for the feature set 422(2) in any technicallyfeasible fashion.

At step 512, for each frame 210(x), the ensemble prediction engine 490configures the prediction engine 192(1) to compute a first frame score434(x) based on the associated set of values for the feature set 422(1)and the perceptual quality model 190(1). At step 514, for each frame210(x), the ensemble prediction engine 490 configures the predictionengine 192(2) to compute a second frame score 434(x) based on theassociated set of values for the feature set 422(2) and the perceptualquality model 190(2).

At step 514, for each frame 210(x), the aggregation engine 440 computesthe aggregated frame score 454 based on the first frame score 434(x) andthe second frame score 434(x). At step 516, the temporal pooling engine194 computes the quality score 184 of the re-constructed target 144based on the aggregated frame scores 454 for the frames 210. Thetemporal pooling engine 194 may perform any number and type of temporalpooling operations to aggregate the frame scores 434. For instance, insome embodiments, the temporal pooling engine 194 determines the qualityscore 184 of the re-constructed target 144 based on the arithmetic meanof the frame scores 434 for the frames 210 included in there-constructed target 144.

The temporal pooling engine 194 then transmits or displays the qualityscore 184 of the re-constructed target 144 for evaluation of streamedvideo content. The method 500 then terminates. Advantageously, asdescribed in greater detail in conjunction with FIG. 3 , any number ofthe quality scores 184 may be aggregated to generate an aggregatequality score as part of evaluating the visual quality of streamed mediacontent.

In sum, the disclosed techniques may be used to efficiently and reliablypredict quality scores for re-constructed video content. During atraining phase, a training engine generates a perceptual quality modelthat maps values for features to a quality score. Subsequently, during aprediction phase, a prediction engine computes per-frame quality scoresfor re-constructed video content based on the perceptual quality modeland values for the features of the re-constructed video content. Atemporal pooling engine then composites the per-frame quality scoresinto a single quality score. Notably, during both the training phase andthe prediction phase, a temporal extractor computes per-frame values formultiple temporal features, where each temporal feature is associatedwith a different scale.

The temporal extractor computes a per-frame value for a specifictemporal feature based on re-constructed video content, the videocontent, and the scale associated with the temporal feature. To computethe per-frame value for a particular frame, the temporal extractorcomputes a frame difference matrix specifying differences in luminancebetween the frame and an adjacent frame. The temporal extractor thendown-samples the frame difference matrix based on the scale to generatea scaled frame difference matrix. Subsequently, the temporal extractorperforms temporal filtering operations and local mean-subtractionoperations on the scaled frame difference matrix to compute a firstweighted entropy matrix that is associated with the frame of there-constructed video content. In an analogous fashion, the temporalextractor derives a second weighted entropy matrix that is associatedwith the corresponding frame of the video content based on the videocontent and the scale. The temporal extractor performs subtractionoperations between the first weighted entropy matrix and the secondweighted entropy matrix to compute an entropy difference matrix.Finally, the temporal extractor performs spatial pooling operations onthe entropy difference matrix to generate a single frame-specific valuefor the feature.

In some embodiments, an ensemble prediction engine computes a qualityscore for re-constructed video content based on two perceptual qualitymodels. Each of the perceptual quality models maps values for adifferent set of features to a different quality score. For each frameincluded in re-constructed video content, an ensemble prediction enginecomputes a first frame score based on the first perceptual quality modeland a first set of values for the first set of features. For each frameincluded in re-constructed video content, the ensemble prediction enginecomputes a second frame score based on the second perceptual qualitymodel and a second set of values for the second set of features. Foreach frame, the ensemble prediction engine then aggregates the firstframe score and the second frame score to generate an aggregated framescore. Finally, the temporal pooling engine generates a single qualityscore based on the aggregated frame scores for the frames included inthe re-constructed video content.

At least one technological improvement relative to prior art is thatcomputing the entropy difference matrices at multiple scales emulatesaspects of how the human visual system processes visual information.Consequently, the temporal features accurately model perceived qualitydegradation attributable to many types of complex temporal videodistortions. Further, computing quality scores based on differentperceptual quality models associated with different sets of featuresreduces the likelihood that increasing the total number of featuresresults in inaccuracies attributable to overfitting.

1. In some embodiments, a computer-implemented method comprisesgenerating a frame difference matrix based on a first frame and a secondframe that are included in re-constructed video content; computing afirst entropy matrix based on the frame difference matrix and a firstscale; computing a first value for a first temporal feature based on thefirst entropy matrix and a second entropy matrix associated with bothvideo content from which the re-constructed video content is derived andthe first scale; and computing a quality score for the re-constructedvideo content based on the first value, a second value for a secondtemporal feature associated with a second scale, and a machine learningmodel trained using a plurality of subjective quality scores, whereinthe quality score indicates a level of visual quality associated withstreamed video content.

2. The computer-implemented method of clause 1, wherein computing thefirst entropy matrix comprises performing one or more down-samplingoperations on the frame difference matrix based on the first scale togenerate a scaled frame difference matrix; performing one or morespatial filtering operations on the scaled frame difference matrix togenerate a local mean-subtracted matrix; and performing one or moreconditioning operations on the local mean-subtracted matrix.

3. The computer-implemented method of clauses 1 or 2, wherein computingthe first value for the first temporal feature comprises performing asubtraction operation between a first entropy value included in thefirst entropy matrix and a second entropy value included in the secondentropy matrix.

4. The computer-implemented method of any of clauses 1-3, whereincomputing the quality score comprises determining a first frame qualityscore associated with the first frame based on a plurality of values fora plurality of temporal features, one or more values for one or morespatial features, and the machine learning model, wherein the pluralityof values includes the first value and the second value; and performingone or more temporal pooling operations between the first frame qualityscore and a second frame quality score associated with the second frame.

5. The computer-implemented method of any of clauses 1-4, whereinperforming the one or more temporal pooling operations comprisesperforming at least one of a linear low pass operation and a non-linearrank-order weighting operation on the first frame quality score and thesecond frame quality score.

6. The computer implemented method of any of clauses 1-5, furthercomprising computing the one or more values for the one or more spatialfeatures based on the re-constructed video content.

7. The computer-implemented method of any of clauses 1-6, whereingenerating the frame difference matrix comprises, for each of aplurality of pixels, computing a luminance difference between a firstluminance of the pixel in the first frame and a second luminance of thepixel in the second frame.

8. The computer-implemented method of any of clauses 1-7, furthercomprising training the machine learning model based on one or moremachine learning algorithms and the plurality of subjective qualityscores, wherein each subjective quality score included in the pluralityof subjective quality scores is associated with different re-constructedtest video content.

9. The computer-implemented method of any of clauses 1-8, wherein theone or more machine learning algorithms comprises at least one of asupport vector machine algorithm and an artificial neural networkalgorithm.

10. In some embodiments, a computer-readable storage medium includesinstructions that, when executed by a processor, cause the processor toperform the steps of generating a first scaled frame difference matrixbased on a first frame, a second frame, and a first scale, wherein thefirst frame and the second frame are included in re-constructed videocontent; generating a second scaled frame difference matrix based on athird frame, a fourth frame, and the first scale, wherein the thirdframe and the fourth frame are included in video content from which there-constructed video content is derived; computing a first entropymatrix based on the first scaled frame difference matrix; computing afirst value for a first temporal feature based on the first entropymatrix and a second entropy matrix associated with the second scaledframe difference matrix; and computing a quality score for there-constructed video content based on the first value, a second valuefor a second temporal feature associated with a second scale, and amachine learning model trained using a plurality of subjective qualityscores, wherein the quality score indicates a level of visual qualityassociated with streamed video content.

11. The computer-readable storage medium of clause 10, wherein computingthe first entropy matrix comprises performing one or more spatialfiltering operations on the first scaled frame difference matrix togenerate a local mean-subtracted matrix; and performing one or moreconditioning operations on the local mean-subtracted matrix.

12. The computer-readable storage medium of clauses 10 or 11, whereinperforming the one or more spatial filtering operations comprisesapplying a spatial isotropic Gaussian filter to the first scaled framedifference matrix.

13. The computer-readable storage medium of any of clauses 10-12,wherein computing the quality score comprises determining a first framequality score associated with the first frame based on a plurality ofvalues for a plurality of temporal features, one or more values for oneor more spatial features, and the machine learning model, wherein theplurality of values includes the first value and the second value; andperforming one or more temporal pooling operations between the firstframe quality score and a second frame quality score associated with thesecond frame.

14. The computer-readable storage medium of any of clauses 10-13,wherein performing the one or more temporal pooling operations comprisesperforming at least one of a linear low pass operation and a non-linearrank-order weighting operation on the first frame quality score and thesecond frame quality score.

15. The computer-readable storage medium of any of clauses 10-14,wherein the one or more spatial features comprise at least one of anadditive impairment measure feature, a blind or referenceless imagespatial quality evaluator feature, and a visual information fidelityfeature.

16. The computer-readable storage medium of any of clauses 10-15,wherein generating the first scaled frame difference matrix comprisesfor each of a plurality of pixels, computing a luminance differencebetween a first luminance of the pixel in the first frame and a secondluminance of the pixel in the second frame to generate a framedifference matrix; and performing one or more down-sampling operationson the frame difference matrix based on the first scale.

17. The computer-readable storage medium of any of clauses 10-16,wherein a first subjective quality score included in the plurality ofsubjective quality scores is associated with one or more human-observedvisual quality scores for re-constructed test video content.

18. The computer-readable storage medium of any of clauses 10-17,further comprising training the machine learning model based on theplurality of subjective quality scores and at least one of a supportvector machine algorithm and an artificial neural network algorithm.

19. In some embodiments a system comprises a memory storinginstructions; and a processor that is coupled to the memory and, whenexecuting the instructions, is configured to generate a frame differencematrix based on a first frame and a second frame that are included inre-constructed video content; perform one or more down-samplingoperations on the frame difference matrix based on a first scale togenerate a first scaled frame difference matrix; compute a first entropymatrix based on the first scaled frame difference matrix; compute afirst value for a first temporal feature based on the first entropymatrix and a second entropy matrix associated with both video contentfrom which the re-constructed video content is derived and the firstscale; perform one or more down-sampling operations on the framedifference matrix based on a second scale to generate a second scaledframe difference matrix; and compute a quality score for there-constructed video content based on the first value, a second valuefor a second temporal feature associated with the second scaled framedifference matrix, and a machine learning model trained using aplurality of subjective quality scores, wherein the quality scoreindicates a level of visual quality associated with streamed videocontent.

20. The system of clause 19, wherein the processor is configured tocompute the first entropy matrix by performing one or more spatialfiltering operations on the first scaled frame difference matrix togenerate a local mean-subtracted matrix; and performing one or moreconditioning operations on the local mean-subtracted matrix.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present invention andprotection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a ““module” or“system.” Furthermore, aspects of the present disclosure may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine. The instructions, when executed via the processor ofthe computer or other programmable data processing apparatus, enable theimplementation of the functions/acts specified in the flowchart and/orblock diagram block or blocks. Such processors may be, withoutlimitation, general purpose processors, special-purpose processors,application-specific processors, or field-programmable gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A computer-implemented method assessing visualquality of re-constructed video content, the method comprising:computing a first visual quality score indicating a first visual qualitylevel for re-constructed video content based on a first set of valuesfor a first set of features and a first model that associates the firstset of values with the first visual quality score; and determining anoverall visual quality score for the re-constructed video content basedon the first visual quality score and a second visual quality score thatis computed based on a second set of values for a second set of featuresand a second model that is different from the first model, wherein thesecond visual quality score indicates a second visual quality level forthe re-constructed video content, and the second model associates thesecond set of values with the second visual quality score.
 2. Thecomputer-implemented method of claim 1, wherein the first model isgenerated via a first machine learning algorithm.
 3. Thecomputer-implemented method of claim 1, wherein the second model isgenerated via a first machine learning algorithm associated with thefirst model or a second machine learning algorithm.
 4. Thecomputer-implemented method of claim 1, wherein the first set offeatures and the second set of features comprise different subsets of anoverall set of features.
 5. The computer-implemented method of claim 1,wherein the first visual quality score and the second visual qualityscore are associated with a first frame included in the re-constructedvideo content, and determining the overall visual quality scorecomprises: determining a first aggregated visual quality scoreassociated with the first frame based on the first visual quality scoreand the second visual quality score; and performing one or more temporalpooling operations on the first aggregated visual quality score and asecond aggregated visual quality score associated with a second frameincluded in the re-constructed video content to compute the overallvisual quality score.
 6. The computer-implemented method of claim 5,wherein determining the first aggregated visual quality score comprisescomputing an average or a weighted average of the first visual qualityscore and the second visual quality score.
 7. The computer-implementedmethod of claim 5, wherein determining the first aggregated visualquality score comprises associating the first visual quality score andthe second visual quality score with the first aggregated visual qualityscore based on a third model that is generated via a first machinelearning algorithm associated with the first model or a second machinelearning algorithm.
 8. The computer-implemented method of claim 1,wherein the first set of features includes at least one temporal featureand at least one spatial feature.
 9. The computer-implemented method ofclaim 1, further comprising generating the first model based on a firstmachine learning algorithm and a plurality of subjective scores, whereineach subjective score included in the plurality of subjective scores isassociated with different re-constructed test video content.
 10. Thecomputer-implemented method of claim 9, further comprising generatingthe second model based on the plurality of subjective scores.
 11. One ormore non-transitory computer-readable storage media includinginstructions that, when executed by one or more processors, cause theone or more processors to perform the steps of: computing a first visualquality score indicating a first visual quality level for re-constructedvideo content based on a first set of values for a first set of featuresand a first model that associates the first set of values with the firstvisual quality score; and determining an overall visual quality scorefor the re-constructed video content based on the first visual qualityscore and a second visual quality score that is computed based on asecond set of values for a second set of features and a second modelthat is different from the first model, wherein the second visualquality score indicates a second visual quality level for there-constructed video content, and the second model associates the secondset of values with the second visual quality score.
 12. The one or morenon-transitory computer-readable storage media of claim 11, wherein thefirst model is generated via a first machine learning algorithm.
 13. Theone or more non-transitory computer-readable storage media of claim 11,wherein the second model is generated via a first machine learningalgorithm associated with the first model or a second machine learningalgorithm.
 14. The one or more non-transitory computer-readable storagemedia of claim 11, wherein the first set of features and the second setof features comprise different subsets of an overall set of features.15. The one or more non-transitory computer-readable storage media ofclaim 11, wherein the first visual quality score and the second visualquality score are associated with a first frame included in there-constructed video content, and determining the overall visual qualityscore comprises: determining a first aggregated visual quality scoreassociated with the first frame based on the first visual quality scoreand the second visual quality score; and performing one or more temporalpooling operations on the first aggregated visual quality score and asecond aggregated visual quality score associated with a second frameincluded in the re-constructed video content to compute the overallvisual quality score.
 16. The one or more non-transitorycomputer-readable storage media of claim 15, wherein performing the oneor more temporal pooling operations comprises performing at least one ofa linear low pass operation, a non-linear rank-order weightingoperation, or an arithmetic mean operation on the first aggregatedvisual quality score and a second aggregated visual quality score. 17.The one or more non-transitory computer-readable storage media of claim11, wherein the first set of features includes a plurality of temporalfeatures, and each temporal feature included in the plurality oftemporal features is associated with a different scale.
 18. A system,comprising: one or more memories storing instructions; and one or moreprocessors that are coupled to the one or more memories and, whenexecuting the instructions, are configured to: compute a first visualquality score indicating a first visual quality level for re-constructedvideo content based on a first set of values for a first set of featuresand a first model that associates the first set of values with the firstvisual quality score; and determine an overall visual quality score forthe re-constructed video content based on the first visual quality scoreand a second visual quality score that is computed based on a second setof values for a second set of features and a second model that isdifferent from the first model, wherein the second visual quality scoreindicates a second visual quality level for the re-constructed videocontent, and the second model associates the second set of values withthe second visual quality score.